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Abstract 


This  document  is  a  guidebook  for  conducting  a  Measurement  and  Analysis  Infrastructure 
Diagnostic  (MAID)  evaluation.  The  MAID  method  is  a  criterion-based  evaluation  method  that 
is  used  to  assess  the  quality  of  an  organization’s  data  and  the  information  generated  from 
that  data. 

The  method  is  organized  into  four  phases:  (1)  Collaborative  Planning,  (2)  Artifact  Evaluation, 
(3)  On-site  Evaluation,  and  (4)  Report  Results.  Using  the  MAID  evaluation  criteria  as  a  guide, 
a  MAID  team  systematically  studies  and  evaluates  an  organization’s  measurement  and 
analysis  practices  by  examining  the  organization’s  data  and  observing  how  the  data  is 
manipulated  during  its  lifecycle,  from  the  collection  of  base  measures  to  the  information 
provided  to  decision  makers. 

The  outcome  of  a  MAID  evaluation  is  a  detailed  report  of  an  organization’s  strengths  and 
weaknesses  in  measurement  and  analysis. 
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About  This  Document 


Introduction 


Intended  audience 


How  this  document 
is  organized 


This  section  describes  this  document  and  how  it  is  organized. 


Intended  readers  include  all  people  who  will  be  involved  in  a 
Measurement  and  Analysis  Diagnosis  (MAID)  effort.  This  includes 
those  conducting  the  MAID  evaluation  and  personnel  from  the 
organization  undergoing  the  evaluation. 


This  document  begins  with  a  Method  Overview  section  that  provides 
an  introduction  to  the  MAID  method.  This  overview  is  followed  by  the 
method  description. 

The  method  description  is  hierarchically  organized  in  the  following 
way: 

Phase  X  Where  X  =  1,  2,  3,  or  4 


StageX.l  StageX.2 


Activity  X.1.1  Activity  X.1.2  Activity  X.1.3  ••• 


MAID  is  composed  of  four  phases: 

1.  Collaborative  Planning 

2.  Artifact  Evaluation 

3.  On-site  Evaluation 

4.  Report  Results 

The  phase  description  provides  an  overview  of  the  activities  within 
that  phase. 

The  stage  description  is  a  breakdown  of  a  phase.  In  some  cases  a 
stage  is  further  decomposed  into  activity  descriptions,  and  the  stage 
description  serves  as  an  overview  of  the  activity  descriptions.  In  other 
cases,  further  decomposition  is  unnecessary  and  the  stage  is 
described  in  the  same  way  as  an  activity. 

The  activity  description  is  typically  elucidated  by  the  components 
listed  in  the  next  table. 


Continued  on  next  page 
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How  this  document 
is  organized,  cont. 


Method 

summary 


Component  Explanation 

Activity 

Description 

Provides  an  overview  of  the  activity. 

Who 

participates? 

The  roles  that  need  to  be  filled  for  the  MAID 
activity.  The  individuals  in  these  roles  are 
responsible  for  ensuring  that  the  activity  is 
performed.  However,  in  many  cases,  it  is 
acceptable  if  the  performance  is  delegated  to 
an  associate. 

Information 

needed 

Information  that  is  required  to  conduct  the 
activity.  In  many  cases,  the  information  needed 
is  a  product  from  a  previous  MAID  activity. 

When  this  is  the  case,  the  listed  item  is 
followed  by  the  phase/stage/activity  number  in 
square  brackets  (i.e.,  [#.#.#]). 

Procedure 

A  concise  step-by-step  description  of  what 
happens  and  who  does  it. 

Products 

The  products  that  are  created  by  the  activity. 

Additional 

guidance 

Additional  heuristics  and/or  tips  for  conducting 
the  activity. 

The  method  description  is  followed  by  the  appendices  and  a  brief 
reference  section. 


Appendix  A  provides  a  summary  of  MAID,  showing  the  information 
needed  and  products  produced  by  each  stage  or  activity. 


2  |  CMU/SEI-2010-TR-035 


Related 

documents 


How  you  can 
provide  feedback 


A  companion  document  describes  the  evaluation  criteria  that  are 
used  during  a  MAID.  This  document  is  referred  to  often  within  the 
MAID  method: 

Measurement  and  Analysis  Infrastructure  Diagnostic  (MAID) 
Evaluation  Criteria,  Version  1.0  [SEMA  2009] 

The  following  document  describes  the  motivation  and  rationale  for 
MAID: 

Can  You  Trust  Your  Data?  Establishing  the  Need  for  a 
Measurement  and  Analysis  Infrastructure  Diagnostic  [Kasunic 
2008] 


We  are  interested  in  your  ideas  for  improving  the  MAID  method  and 
its  application.  If  you  have  suggestions  or  questions,  please  send 
email  to  info@sei.cmu.edu. 
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Method  Overview 


Introduction  How  reliable  is  the  data  that  organizations  collect  and  use?  How 

useful  and  effective  is  the  information  that  is  generated  from  that 
data? 

Given  the  importance  of  measurement  and  analysis  in  decision 
making,  organizations  need  to  ensure  their  data,  analyses,  and 
corresponding  reports  are  of  high  quality  and  meet  the  information 
needs  of  practitioners  and  managers.  Yet,  organizations  often  have 
no  method  for  evaluating  the  quality  of  their  measurement  and 
analysis  infrastructure  and  have  very  little  trust  in  their  own  data. 

Ensuring  information  quality  is  a  challenge  for  most  organizations — 
partly  because  they  may  not  be  fully  aware  of  their  own  data  quality 
levels.  Without  this  information,  they  cannot  know  the  full  business 
impact  of  poor  or  unknown  data  quality  or  how  to  begin  addressing  it. 


What  is  the  MAID  The  Measurement  and  Analysis  Infrastructure  Diagnostic  (MAID) 
method?  method  is  a  criterion-based  evaluation  method  that  assesses  the 

quality  of  an  organization’s  data  and  the  information  generated  from 
that  data.  The  method  is  organized  into  four  phases,  as  illustrated  in 
Figure  1 . 


Figure  1:  The  MAID  Method 
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The  MAID  Using  the  MAID  evaluation  criteria  as  a  guide,  a  MAID  team 

context  systematically  studies  and  evaluates  an  organization’s  measurement 

and  analysis  practices  by  examining  the  organization’s  data  and 
observing  how  the  data  is  manipulated  during  its  lifecycle,  from  the 
collection  of  base  measures  to  the  information  provided  to  decision 
makers.  (See  Figure  2.) 
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Figure  2:  Measurement  Information  Stream  From  Its  Source 


MAID  objectives  A  MAID  evaluation  is  conducted  to  meet  two  objectives: 

1 .  Evaluate  data  quality  and  compare  an  organization’s  current 
measurement  and  analysis  practices  against  a  defined  set  of 
criteria  that  are  based  on  best  practices. 

With  respect  to  this  objective,  some  of  the  questions  that  the 
organization  needs  to  answer  include  the  following: 

•  Are  we  doing  the  right  things  in  terms  of  measurement  and 
analysis? 

•  Are  information  needs  being  met?  How  well? 

•  How  good  is  our  data? 

•  How  well  does  the  information  we  generate  support  effective 
decision  making? 

•  What  kind  of  value  is  our  measurement  program  really 
providing  to  the  organization  and  its  stakeholders? 

Continued  on  next  page 
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MAID  objectives, 
Continued 


MAID  is  a 
collaboration 


2.  Make  recommendations  for  improvement. 

Questions  that  are  addressed  through  this  objective  include  the 

following: 

•  Are  there  hidden  quality  issues  that  are  affecting  the  accuracy 
and  reliability  of  measurement  information  reported  to 
decision  makers? 

•  Are  there  opportunities  to  improve  the  quality  of  the  M  &  A 
information?  What  are  they? 

•  How  can  identified  gaps  or  weaknesses  be  addressed? 

•  Are  there  opportunities  to  improve  the  quality  of  the  M&A 
information?  What  are  they? 

•  What  M&A  practices  are  “best  practices”  and  should  be 
encouraged  across  the  organization? 

How  can  we  prepare  for  achieving  higher  process  maturity? 


A  MAID  evaluation  is  conducted  through  a  collaboration  between  an 
evaluation  team  and  members  of  the  client  organization  that  has 
requested  a  MAID  (see  Figure  3). 

Although  the  MAID  team  interacts  directly  with  many  individuals  from 
the  client  organization,  much  of  the  communication — especially 
during  the  first  two  phases  of  MAID — is  facilitated  by  the  client  point  of 
contact  (POC). 


Figure  3:  MAID  Team  Collaborates  With  Organization  to  Produce  Result 
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Key  roles 


The  following  table  lists  the  key  roles  that  need  to  be  filled  for  a  MAID 
evaluation. 


Role 

Description 

Team  lead 

•  leader  of  the  evaluation  team 

•  M  &  A  expert 

•  point  of  contact  (POC)  for  communication 
between  client  and  MAID  evaluation  team 

•  responsible  for  all  administrative  and 
organization  activities  associated  with  a  MAID 

Team  members 

•  M  &  A  experts  who  participate  as  MAID 
evaluators 

•  participate  in  MAID  activities  based  on  needs 
expressed  by  the  MAID  team  lead 

Sponsor 

•  senior  client  manager  who  has  authorized  the 
MAID  evaluation 

•  participates  in  MAID  planning  and  is  the 
recipient  of  the  MAID  results 

Client  POC 

•  POC  for  communication  between  the  client 
organization  and  the  MAID  team  lead 

•  coordinates  planning  and  scheduling  of  MAID 
activities  for  the  client  organization 

•  facilitates  delivery  of  M  &  A  artifacts  to  the 
team  lead 

•  orchestrates  all  activities  to  facilitate  the 

conduct  of  MAID  at  the  client’s  site 

Client  SME 

•  client  M  &  A  subject  matter  experts  (SMEs) 
who  are  knowledgeable  about  the  M  &  A 
activities,  tools,  and  work  products  that  are 
within  the  defined  MAID  scope 

•  answers  questions  about  the  M  &  A 
documentation  submitted  as  part  of  the  MAID 
scope 

•  participates  in  MAID  interviews 

Questionnaire 

respondents 

•  respond  to  questions  or  questionnaires  that 
are  included  as  part  of  the  MAID  scope 

Stakeholders 

•  client  individuals  who  are  interested  in  the 

results  of  a  MAID  evaluation 
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MAID  evaluation 
team 


Team  composition 
and  skill  set 


The  MAID  evaluation  team  can  be  composed  of  (a)  individuals  from 
the  client  organization  if  the  individuals  possess  M  &  A  expertise, 

(b)  external  M  &  A  experts  who  are  contracted  by  the  organization  to 
perform  the  MAID  evaluation,  or  (c)  a  combination  of  a  and  b. 


Conducting  a  MAID  evaluation  involves  making  judgments  about 
aspects  of  M  &  A  that  include  (but  are  not  limited  to) 

•  measurement  definition  approaches 

•  data  quality  attributes 

•  data  storage  and  access  mechanisms 

•  M  &  A  processes  and  procedures 

•  data  analysis  techniques 

•  M  &  A  information  design 

Therefore,  it  is  important  that  the  MAID  evaluation  team  is  composed 
of  individuals  with  the  knowledge  and  skills  needed  to  make  these 
judgments. 

Team  members  should  also  possess 

•  organizational  skills  (to  perform  a  significant  amount  of 
planning,  document  handling,  and  record-keeping) 

•  interviewing  and  effective  listening  skills 

•  effective  writing  skills 

Phase  2:  Artifact  Evaluation  involves  conducting  interviews.  Having  a 
team  member  who  is  effective  at  recording  the  interviews  is  essential. 
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MAID  evaluation 
criteria 


The  MAID  method  uses  a  set  of  criteria  that  draws  upon,  synthesizes, 
and  organizes  well-established  best  M  &  A  practice  guidance  that  is 
already  embodied  in  existing  models,  standards,  methods,  and 
applicable  best  practices. 

The  MAID  criteria  were  published  in  Measurement  and  Analysis 
Infrastructure  Diagnostic  (MAID)  Evaluation  Criteria,  Version  1.0 
[SEMA  2009].  The  criteria  are  organized  into  five  categories, 
described  in  Table  1. 


Table  1 :  Major  topics  covered  in  each  category  of  MAID  criteria 


Criteria  Categories 

Major  Topics 

M  &  A  Planning 

• 

• 

• 

• 

• 

Identifying  M  &  A  resource  needs  and  training  needs 
Identifying  M  &  A  information  needs 

Identifying  and  defining  measures  and  measurement 
indicators  that  address  needs 

Planning  and  scheduling  M  &  A  activities 

Developing  M  &  A  processes  and  procedures  to  perform 

M  &  A  activities 

Data  Collection  and 
Storage 

• 

• 

• 

Collecting  M  &  A  data  that  address  information  needs 

Ensuring  completeness  and  accuracy  of  the  recorded  data 

Ensuring  that  data  are  stored  securely 

Data  Analysis 

• 

• 

• 

• 

Preparing  the  data  for  analysis 

Transforming  the  data  into  tables  and  graphs  that  can  be 
analyzed 

Selecting  the  appropriate  statistical  analysis  approach  to 
analyze  the  data 

Conducting  statistical  analyses  of  the  data 

M  &  A  Reporting 

• 

• 

Understanding  the  M  &  A  expertise  level  of  the  stakeholders 
who  will  use  the  M  &  A  information  to  support  decision 
making 

Transforming  the  data  analysis  results  into  M  &  A 
information  that  best  facilitates  comprehension  by  the 
stakeholders  who  will  use  the  information  to  support 
decision  making 

Criteria  for  All  M  &  A 

Process  or  Procedure 

Documentation 

• 

Writing  effective  M  &  A  documentation 
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MAID  record¬ 
keeping 

Accurate  recordkeeping  is  a  critical  facet  of  the  MAID  method.  Each 
MAID  criterion  is  rated  and  rationale  is  provided  for  each  rating. 
References  to  artifacts,  interviews,  and  observations  are  documented 
to  ensure  traceability  to  the  source  of  any  findings. 

Organizing  the  criteria  into  a  spreadsheet  with  additional  columns 
reserved  for  ratings,  references,  and  other  comments  provides  a 
simple  software  support  tool  to  assist  the  evaluation  team.1 

MAID  is  tailored  to 

meet  needs 

MAID  is  not  a  one-size-fits-all  method.  The  approach  is  highly 
tailorable  based  on  the  specific  objectives  of  the  client  organization 
that  has  requested  a  MAID  evaluation. 

An  organization  may  choose  to  select  a  large  scope  (e.g.,  one  that 
encompasses  multiple  projects  including  measurement  roll-up  for 
reporting  to  the  executive  level)  or  a  smaller  scope  that  addresses 
only  one  or  two  projects. 

MAID  can  address  the  entire  M  &  A  life  cycle  (Planning,  Data 

Collection  and  Storage,  Data  Analysis,  and  M  &  A  Reporting). 

However,  an  organization  may  choose  to  focus  on  one  or  several  of 
these  phases  of  the  life  cycle. 

Other  tailoring  options  are  available  at  the  discretion  of  the  client 
organization. 

Communication 

The  need  for  communication  is  not  always  spelled  out  in  MAID 
activities.  However,  effective  communication  is  critical  to  the  MAID 

method. 

The  MAID  team  leader  and  client  POC  must  ensure  that  all 
stakeholders  are  kept  informed  throughout  the  execution  of  MAID. 

The  SEI  is  collaborating  with  the  Universidad  Carlos  III  de  Madrid  to  develop  a  web-based  software 
support  tool  for  MAID. 
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1  Collaborative  Planning 


1.  Collaborative 
planning 

_  2.  Artifact  _  3.  On-site  _  4.  Report 

evaluation  evaluation  results 

Introduction 

During  Phase  1 :  Collaborative  Planning,  the  MAID  team  and  the  client 
organization  plan  and  prepare  for  the  MAID. 

The  evaluation  team  leader  collects  information  from  the 
organization’s  sponsor  to  ensure  that  MAID  objectives  align  with  and 
support  the  business  goals  of  the  organization. 

Before  you 
begin 

Before  beginning  this  phase,  the  organization  sponsor  should  be  able 
to  describe  the  organization’s  business  needs  and  how  a  MAID 
evaluation  will  address  those  needs. 

Stages 

The  stages  included  in  this  phase  are 

1.1  Establish  Scope 

1 .2  Establish  Roles  and  Expectations 

1.3  Develop  Plan  and  Schedule 
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1.1  Establish  Scope 


Overview 


Activities  during 
this  stage 


T radeoffs  are 
involved 


During  this  stage  the  sponsor  works  with  the  MAID  team  leader  to 
identify  the  specific  areas  to  focus  on  during  the  evaluation.  To 
accomplish  this,  the  sponsor  must  take  into  account 

•  the  prioritized  business  needs  of  the  organization 

•  the  objectives  for  the  MAID  evaluation 

•  cost  and  time  constraints 

The  objectives  that  motivate  a  MAID  intervention  must  be  well 
understood  and  balanced  with  the  constraints  so  that  the  appropriate 
participants,  tailoring  decisions,  and  MAID  outputs  can  be 
determined. 


This  stage  is  composed  of  the  following  activities: 

1.1.1  Determine  MAID  Objectives 

1.1.2  Determine  Constraints 

1.1.3  Determine  the  MAID  Scope 


The  intention  is  to  find  the  right  balance  between  objectives  and 
constraints.  It  is  sometimes  necessary  to  iterate  between  activities 
1.1.1  (Determine  MAID  Objectives)  and  1.1.2  (Determine  Constraints) 
in  order  to  achieve  1.1.3  (Determine  the  MAID  Scope). 


Constraints 


Objectives 


>»» 


MAID 

Scope 


Figure  4:  Balancing  objectives  and  constraints 
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1.1.1  Determine  MAID  Objectives 


Activity 

description 


Examples  of  MAID 
objectives 


Who 

participates? 


Information 

needed 


Procedure 


The  purpose  of  this  activity  is  to  develop  a  shared  understanding  of 
the  MAID  objectives  among  the  organization  sponsors,  MAID 
evaluation  team,  and  all  other  MAID  participants. 


The  business  needs  of  the  organization  drive  the  requirements  for  a 
MAID  evaluation.  An  organization  conducts  the  evaluation  to 

•  understand  the  degree  of  data  and  M  &  A  information  quality  of 
the  organization 

•  identify  risk  areas  within  its  M  &  A  program 

•  improve  its  M  &  A  practices  in  a  project,  set  of  projects,  or  across 
an  entire  organization 

•  benchmark  its  M  &  A  practices  across  the  organization 

•  benchmark  its  M  &  A  practices  with  other  organizations 

•  obtain  guidance  for  expanding  or  evolving  its  M  &  A  program 

•  Sponsor 

•  Team  lead 


Description  of  the  organization’s  business  goals  or  business  needs 


The  table  below  describes  the  key  tasks  of  this  activity. 


Step 

Who? 

What  happens? 

1 

Team  lead 

Identifies  sponsor  and  other  stakeholders. 

2 

Team  lead 

Establishes  communication  with  sponsor. 

3 

Team  lead, 
Sponsor 

Team  lead  documents  the  business  goals 
and  the  MAID  objectives  provided  by  the 
sponsor. 

4 

Team  lead, 
Sponsor 

Verifies  alignment  of  MAID  objectives  with 
business  goals. 
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Products 


List  of  MAID  objectives 


Additional 

Guidance 


•  In  some  cases,  the  sponsor  will  have  a  representative  (such  as 
the  client  POC)  work  on  the  details  of  the  initial  planning.  This  is 
acceptable  as  long  as  the  sponsor  is  kept  informed  of  those 
details  along  the  way  and  buys  into  the  agreements  that  are 
negotiated  and  documented. 

•  The  sponsor  must  acknowledge  his  or  her  agreement  with  the 
MAID  objectives.  One  way  to  do  so  is  to  sign  the  documented 
objectives. 
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1.1.2  Determine  Constraints 


Activity 

description 


Who 

participates? 


Information 

needed 


Procedure 


The  constraints  under  which  a  MAID  evaluation  is  conducted  are 
typically  posed  by  such  factors  as 

•  cost 

•  schedule 

•  personnel  availability 

During  this  activity,  the  sponsor  and  the  evaluation  team  lead  discuss 
the  constraints  and  how  this  may  limit  the  scope  of  the  evaluation. 


•  Sponsor 

•  Team  lead 


•  List  of  MA  ID  objectives  [1.1.1] 

•  Estimates  of  personnel  availability 


The  table  below  describes  key  tasks  of  this  activity. 


Step 

Who? 

What  happens? 

1 

Sponsor,  Team  lead 

Establish  high-level  cost  and  schedule 
constraints. 

2 

Sponsor,  Team  lead 

Determine  which  organizational  entities 
are  candidates  for  inclusion  in  the 

MAID  evaluation. 

3 

Sponsor,  Team  lead 

Negotiate  constraints  and  objectives  to 
ensure  feasibility. 

4 

Team  lead 

Documents  negotiated  constraints  to 
be  met  and  revisions  to  the  original 
objectives  based  on  the  results  of  this 
activity. 
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Products 


List  of  constraints  to  be  met  (accompanied  by  rationale  for 
decisions  made) 

List  of  personnel  availability 

List  of  revised  MAID  objectives  (based  on  the  results  of  this  activity) 
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1 .1 .3  Determine  MAID  Scope 


Activity 

description 


Who 

participates? 


Information 

needed 


The  MAID  scope  consists  of  two  parts:  (1)  the  scope  based  on 
categories  of  MAID  evaluation  criteria  that  will  be  used  and  (2)  the 
organizational  scope. 

Five  categories  of  evaluation  criteria  can  be  used  during  a  MAID 
evaluation: 

1.  M&APIanning 

2.  Data  Collection  and  Storage 

3.  Data  Analysis 

4.  M  &  A  Reporting 

5.  M  &  A  -  General 

In  some  cases,  the  client  may  want  only  a  subset  of  these  categories 
to  be  evaluated  instead  of  all  five.  This  is  especially  likely  in 
organizations  that  are  just  starting  their  M  &  A  programs;  these 
organizations  may  want  to  limit  the  scope  to  an  evaluation  of  their  M 
&  A  planning  progress.  A  decision  about  which  categories  to  include 
should  be  made  based  on  the  objectives  and  constraints. 

The  organizational  scope  refers  to  the  identification  of  the 
organizational  entities  that  will  be  included  as  part  of  the  MAID 
evaluation.  A  full  organizational  scope  would  include  the  organization 
and  all  projects  within  the  organization.  A  partial  scope  would  include 
a  subset  of  the  full  organizational  scope. 


•  Sponsor 

•  Team  lead 


•  List  of  revised  MAID  objectives  [1 .1 .2] 

•  List  of  constraints  to  be  me  [1 . 1 .2] 
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Procedure 


The  table  below  describes  the  key  tasks  of  this  activity. 


Products 


Additional 

guidance 


Step 

Who? 

What  happens? 

1 

Sponsor,  Team  lead 

Selects  the  organizational  entities  that 
will  be  included  in  the  MAID  evaluation. 

2 

Sponsor,  Team  lead 

For  each  entity  selected  in  step  1, 
selects  the  MAID  criteria  categories 
that  will  be  used  to  evaluate  each 
entity. 

3 

Team  lead 

Documents  the  results  of  steps  1  and  2 
and  provides  rationale  for  the  decision. 

MAID  scope  statement  that  includes 

•  identification  of  organizational  entities  that  will  be  included  in  the 
MAID  evaluation 

•  identification  of  MAID  criteria  categories  that  will  be  used  for  each 
organizational  entity 

•  rationale  for  decisions  made 


The  scope  identified  during  this  early  stage  of  MAID  is  expected  to  be 
high  level  and  not  very  detailed.  During  subsequent  planning,  the 
scope  may  be  modified.  For  example,  it  might  change  after  detailed 
estimates  are  made  about  the  effort  involved  in  carrying  out  the  MAID 
evaluation. 
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1.2  Establish  Roles  and  Expectations 


Overview 


Key  roles 


Once  the  scope  for  the  MAID  evaluation  is  established,  the 
appropriate  personnel  are  identified  and  enlisted  to  participate  in  the 
effort.  This  includes  members  of  the  evaluation  team  and  individuals 
from  the  client  organization  undergoing  the  evaluation. 


This  table  lists  the  key  roles  that  need  to  be  filled  during  a  MAID 
evaluation. 


Role 

Description 

Team  lead 

•  leader  of  the  evaluation  team 

•  M  &  A  expert 

•  point  of  contact  (POC)  for  communication 
between  client  and  MAID  evaluation  team 

•  responsible  for  all  administrative  and 
organization  activities  associated  with  a 

MAID 

Team  members 

•  M  &  A  experts  who  participate  as  MAID 
evaluators 

•  participate  in  MAID  activities  based  on  needs 
expressed  by  MAID  team  lead 

Sponsor 

•  senior  client  manager  who  has  authorized 
the  MAID  evaluation 

•  participates  in  MAID  planning  and  is  the 
recipient  of  the  MAID  results 

Client  POC 

•  POC  for  communication  between  the  client 
organization  and  the  MAID  team  lead 

•  coordinates  planning  and  scheduling  of  MAID 
activities  for  the  client  organization 

•  facilitates  delivery  of  M  &  A  artifacts  to  the 
team  lead 

•  orchestrates  all  activities  to  facilitate  the 

conduct  of  MAID  at  the  client’s  site 

Table  continues  on  next  page 
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Key  roles,  cont. 


Who 

participates? 


Procedure 


Client  SME  •  client  M  &  A  subject  matter  experts 

(SMEs)  who  are  knowledgeable  about 
the  M  &  A  activities,  tools,  and  work 
products  that  are  within  the  defined 
MAID  scope 

•  answers  questions  about  the  M  &  A 
documentation  submitted  as  part  of  the 
MAID  scope 


•  participates  in  MAID  interviews 


Questionnaire 

•  responds  to  questions  or 

respondents 

questionnaires  that  are  included  as 

part  of  the  MAID  scope 

Stakeholders 

•  client  individuals  who  are  interested  in 

the  results  of  a  MAID  evaluation 

•  Team  lead 

•  Client  POC 


The  table  below  describes  the  key  tasks  of  this  activity. 


Step 

Who? 

What  happens? 

la 

Team  lead 

Based  on  the  established  scope,  determines  the 
appropriate  size  of  the  MAID  team.  Obtains 
additional  resources  as  required. 

Records  availability  information  for  each  team 
member  (e.g.,  vacation  days,  days  not  available, 

%  time  available  for  the  MAID  evaluation). 

1b 

Client 

POC 

Identifies  the  client  SME  for  each  of  the  areas  that 
will  be  evaluated  during  the  MAID  evaluation  (see 
Additional  guidance  on  page  23). 

Table  continues  on  next  page 
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Procedure,  cont. 


Step 

Who? 

What  happens? 

3 

Client  POC 

Communicates  with  each  of  the  client  SMEs 
to  brief  them  on  their  MAID  role  and  how  they 
will  participate  in  the  effort. 

Determines  if  availability  will  be  an  issue. 
Records  availability  information  for  each 

SME.2 

Coordinates  SME  participation  with 
management. 

4 

Client  POC 

Develops  a  List  of  client  SMEs  including 
organizational  role  and  contact  information. 

Delivers  this  list  to  the  team  lead. 

Products  •  List  of  MAID  team  members  with  contact  information  and 

availability  information 

•  List  of  client  SMEs  that  includes  the  following  descriptors 

-  name 

-  contact  information  (phone,  email) 

-  organization/project 

-  organization/project  title 
M  &  A  role  in  organization 

-  M  &  A  SME  area  designation3 

-  availability  information 


Additional  •  SMEs  are  selected  based  on  their  knowledge  of  M  &  A  planning, 

guidance  procedures,  and  practices  for  the  organization  or  for  a  particular 

project  or  role  such  as  a  data  manager  or  data  analyst.  An  SME 
could  also  be  a  preparer  of  measurement  information  for  reports 
or  briefings. 

•  In  some  cases,  there  may  be  multiple  SMEs  for  a  particular  area 
that  will  undergo  MAID  evaluation. 


2  This  information  is  used  as  input  for  developing  a  schedule  of  MAID  activities. 

3  This  is  the  area  in  which  the  individual  has  expertise  or  extensive  knowledge.  The  area  may  be  a 
data  repository  designation  or  it  could  refer  to  specific  M  &  A  planning  or  process/procedure  areas. 
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1.3  Develop  Plan  and  Schedule 


Overview 


Activities  during 
this  stage 


Additional 

guidance 


During  this  stage,  the  MAID  team  lead  works  with  the  client  POC  to 
establish  a  plan  and  schedule  for  conducting  the  MAID  evaluation. 

The  MAID  team  lead  coordinates  with  the  MAID  team  members  to 
obtain  planning/scheduling  inputs  and  constraints  while  the  client 
POC  coordinates  with  individuals  from  the  organization  to  obtain 
inputs  that  are  relevant  to  planning  the  MAID  evaluation. 


This  stage  comprises  the  following  activities: 

1.3.1  Determine  MAID  Outputs 

1.3.2  Determine  MAID  Inputs 

1.3.3  Tailor  Method 

1 .3.4  Determine  Cost  and  Schedule 

1 .3.5  Obtain  Commitment  to  the  Plan 


Planning  is  iterative  in  nature.  Once  the  baseline  plan  is  documented, 
additional  detail  should  be  added  as  it  becomes  available  over  time. 
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1.3.1  Determine  MAID  Outputs 


Activity 

description 

Who 

participates? 


Information 

needed 


Types  of  MAID 
output 


Procedure 


Products 


During  this  activity,  the  specific  outputs  of  the  MAID  evaluation  are 
discussed,  negotiated,  and  agreed  upon. 

•  Client  POC 

•  Team  lead 


•  List  of  revised  MAID  objectives  [1.1.2] 

•  Input  from  sponsor  to  client  POC 


The  results  of  a  MAID  evaluation  can  be  reported  in  several  ways, 
including  one  or  a  combination  of  the  following: 

•  presentation  in  slide  format  of  M  &  A  strengths  and  weaknesses 

•  MAID  survey  results 

•  list  of  MAID  criteria  with  ratings  for  each  criterion  and  rationale  for 
the  rating 

•  recommendations  for  addressing  M  &  A  weaknesses 

•  detailed  report  that  elaborates  the  MAID  findings 


The  table  below  describes  the  key  tasks  of  this  activity. 


Step 

Who? 

What  happens? 

1 

Client  POC, 

Team  lead 

Discuss  the  various  options  and  the  impact  of 
each  on  effort  and  schedule. 

2 

Client  POC 

Solicits  preferences  from  the  sponsor 
regarding  desired  reporting  content  and 
format  for  the  MAID  results. 

3 

Client  POC 

Communicates  desired  output  to  team  lead. 

List  of  outputs  with  descriptions  that  will  be  developed  based  on  the 
results  of  the  MAID  evaluation 
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1.3.2  Determine  MAID  Inputs 


Activity 

description 


Who 

participates? 


Information 

needed 

Procedure 


Product 

Product 

description 


During  this  activity,  all  M  &  A-related  documents,  M  &  A  tools,  and 
data  repositories  included  in  the  MAID  scope  are  identified  and 
cataloged. 


•  Client  SMEs 

•  Client  POC 

•  Team  lead 


List  of  client  SMEs  [1 .2] 


The  table  below  describes  the  key  tasks  of  this  activity. 


Step 

Who? 

What  happens? 

1 

Client  POC 

Contacts  client  SMEs  to  have  them  submit 
their  input  to  the  M  &  A  artifact  list. 

2 

Client  SMEs 

Provides  input  to  the  M  &  A  artifact  list  based 
on  MAID  assignment  area. 

3 

Client  POC 

Sends  the  completed  M  &  A  artifact  inventory 
to  the  team  lead. 

M  &  A  artifact  inventory 

The  M  &  A  artifact  inventory  is  a  list  where  each  record  contains  the 
following  information: 

•  name  of  artifact 

•  filename  of  artifact  (if  applicable) 

•  type  of  file  (if  applicable) 

•  how  the  artifact  is  used  in  the  organization  (e.g.,  M  &  A  planning, 
guidance  for  data  collection  and  storage,  a  data  reporting  artifact) 

•  number  of  pages  (if  a  document) 

•  number  of  records  (if  a  spreadsheet  or  database) 

•  name  of  client  SME  who  is  knowledgeable  about  the  artifact 
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1.3.3  Tailor  Method 


Activity 

description 


Who 

participates? 


Information 

needed 


Procedure 


Products 


The  MAID  method  provides  a  wide  variety  of  options  to  allow  the 
client  organization  to  select  the  features  of  the  method  that  best 
address  the  client’s  business  objectives. 

During  this  activity,  the  client  selects  specific  implementation  options 
based  on  objectives  and  constraints. 


•  Client  POC 

•  Team  lead 


•  List  of  revised  MAID  objectives  [1 .1 .2] 

•  List  of  constraints  to  be  met  accompanied  by  rationale  for 
decisions  made  [1.1.2] 

•  MAID  scope  statement  [1.1.3] 


The  table  below  describes  the  key  tasks  of  this  activity. 


Step 

Who? 

What  happens? 

1 

Client  POC 

Steps  through  MAID  Method  Definition 
Document  (i.e.,  this  document)  to 
determine  candidate  activities  that 
might  need  to  be  tailored. 

2 

Client  POC,  Team 

lead 

Meet  to  discuss  tailoring  options  and 
risks/benefits  of  pursuing  an  option. 

3 

Team  lead 

Documents  tailoring  decisions  and 
submits  them  to  the  client  POC  for 
approval. 

4 

Client  POC 

Reviews  tailoring  decisions  and 
approves. 

Approved  tailoring  decisions 
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Additional 

guidance 


Some  example  areas  where  the  method  can  be  tailored  include 

•  conducting  executive  sessions 

•  selecting  data  collection  approaches,  such  as  whether  to  use 
a  survey  or  not 

•  conducting  site  interviews  of  small  groups  vs.  individuals 

•  adjusting  the  level  of  rigor  applied  to  data  error  checking  by 
MAID  evaluators 

•  determining  the  detail  of  feedback  and  recommendations 
included  when  reporting  MAID  results 
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1.3.4  Determine  Cost  and  Schedule 


Activity 

description 

Who 

participates? 

Information 

needed 


Procedure 


Product 

Additional 

guidance 


An  initial  cost  breakdown  and  schedule  are  developed  for  review  by 
the  client  organization. 


•  Team  lead 

•  Client  POC 


•  List  of  revised  MAID  objectives  [1 .1 .2] 

•  MAID  scope  statement  [1.1.3] 

•  Approved  tailoring  decisions  [1 .3.3] 


The  table  below  describes  the  key  tasks  of  this  activity. 


Step 

Who? 

What  happens? 

1 

Team  lead 

Estimates  the  duration  of  MAID 
activities  as  a  basis  for  deriving  a 
comprehensive  schedule. 

2 

Team  lead,  Client 
POC 

Meet  to  review  the  results  of  step  1 . 

3 

Team  lead 

Estimates  the  cost  for  incidentals 
associated  with  conducting  the  MAID 
evaluation  (e.g.,  travel,  lodging,  meals). 

4 

Team  lead 

Documents  the  results  and  sends  to 

client  POC  for  review  and  feedback. 

5 

Client  POC 

Reviews  cost  and  schedule  estimates 
and  provides  feedback  to  team  lead. 

5 

Team  lead 

If  client  POC  has  change  requests, 
begins  again  at  step  1. 

Approved  cost  and  schedule  estimates  for  MAID  effort 


In  many  organizations,  it  may  be  necessary  to  obtain  guidance  from 
one  of  the  organization’s  financial  analysts. 
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1.3.5  Obtain  Commitment  to  the  Plan 


Activity 

description 

Who 

participates? 


Information 

needed 


Procedure 


Products 


This  activity  involves  authoring  the  plan  and  obtaining  formal 
commitment  to  the  MAID  plan  from  the  sponsor. 


•  Team  lead 

.  Client  POC 

•  Client  sponsor 

•  Client  SMEs 

•  Stakeholders 


•  List  of  revised  MAID  objectives  [1.1.2] 

•  List  of  personnel  a  vailability  [1.1.2] 

•  The  following  parts  of  the  MAID  scope  statement  [1.1.3]: 

-  Identification  of  organizational  entities  that  will  be  included  in 
the  MAID  evaluation 

-  Identification  of  MAID  criteria  categories  that  will  be  used  for 
each  organizational  entity 

•  Approved  cost  and  schedule  estimates  for  MAID  effort  [1 .3.4] 


The  table  below  describes  the  key  tasks  of  this  activity. 


Step 

Who? 

What  happens? 

1 

Team  lead 

Compiles  the  various  inputs  (information 
needed)  and  documents  the  MAID  plan. 

2 

Client  POC 

Reviews  the  plan  and  provides  feedback 
to  the  team  lead. 

3 

Client  sponsor, 
Client  POC, 

Team  lead 

Team  lead  and  client  POC  review  the  plan 
with  sponsor  and  secure  the  sponsor’s 
approval. 

4 

Client  POC 

Provides  the  MAID  plan  to  all  relevant 
stakeholders  for  review. 

Approved  MAID  plan 
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2  Artifact  Evaluation 


1.  Collaborative 
planning 


Introduction 


Before  you 
begin 


Stages 


During  Phase  2:  Artifact  Evaluation,  the  MAID  team  analyzes  the 
client  organization’s  M  &  A  artifacts  that  were  inventoried  during 
Phase  1 :  Collaborative  Planning.  The  analysis  is  based  on  a  pre¬ 
defined  set  of  MAID  criteria  that  were  also  identified  during  Phase  1 
as  part  of  the  scope  definition. 

The  outcome  of  this  phase  includes: 

•  a  rating  for  each  MAID  criterion  that  is  part  of  the  evaluation,, 
with  rationale  for  the  rating 

•  a  set  of  interview  questions  that  will  be  used  during  Phase  3: 
On-site  Evaluation 


Before  beginning  this  phase,  the  organization  sponsor  approves  the 
MAID  plan  that  was  developed  during  Phase  1 :  Collaborative 
Planning. 


The  stages  included  in  this  phase  are: 

2.1  Prepare  for  Artifact  Evaluation 

2.2  Conduct  M  &  A  Artifact  Evaluation 

2.3  Perform  Quality  Audit  of  Results 

2.4  Prepare  for  On-Site  Evaluation 
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2.1  Prepare  for  Artifact  Evaluation 


Activity 

description 


During  this  activity,  the  MAID  team  prepares  to  conduct  the  artifact 
evaluation.  This  involves  obtaining  the  M  &  A  artifacts  from  the  client 
POC,  organizing  the  artifacts,  and  assigning  team  members  to 
evaluate  specific  M  &  A  artifacts. 


Who 

participates? 


Team  lead 
Team  members 
Client  POC 


Information 

needed 


•  List  of  MAID  team  members  with  contact  information  and 
availability  information  [1 .2] 

•  List  of  client  SMEs  [1 .2] 

•  M  &  A  artifact  inventory  [1 .3.2] 

•  Approved  MAID  plan  [1.3.5] 


Procedure 


The  table  below  describes  the  key  tasks  of  this  activity. 


Step 

Who? 

What  happens? 

1 

Client  POC 

Provides  a  copy  of  each  file  listed  in  the 
inventory  to  the  MAID  team  lead. 

2 

Team  lead 

Organizes  the  files  into  a  repository. 

3 

Team  lead 

Assigns  each  file  to  a  MAID  team  member 
for  evaluation. 

Products 


M  &  A  artifacts  received  from  the  client  organization 


Additional  Mechanisms  must  be  put  into  place  to  control  versioning  of  the 

guidance  artifacts.  A  master  copy  of  each  artifact  should  be  stored  away  from 

the  copies  that  will  be  used  during  the  evaluation  as  Team  Members 
will  likely  annotate  the  documents  during  their  review.4 


4  The  notes  made  within  the  artifacts  by  the  MAID  team  members  during  the  evaluation  activity  are 
off  record.  These  might  include,  for  example,  work-in-progress  notes  that  assist  the  reviewers  with 
their  work  but  are  not  relevant  to  the  evaluation  outcome. 
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2.2  Conduct  M  &  A  Artifact  Evaluation 


Activity  During  this  activity,  the  MAID  team  members  conduct  analyses  of 

description  M  &  A  artifacts  that  are  included  in  the  MAID  scope. 

This  activity  describes  the  approach  for  evaluating  a  single  M  &  A 
artifact.  The  activity  would  be  repeated  for  each  M  &  A  artifact  that  is 
included  as  part  of  the  inventory. 


Who 

participates? 


Team  lead 
Team  members 
Client  POC 
Client  SMEs 


Information 

needed 


M  &  A  artifact  inventory  [1 .3.2] 

M  &  A  artifact  received  from  the  client  organization  [2.1  ] 
MAID  criteria  form 


Procedure 


The  table  below  describes  the  key  tasks  of  this  activity. 


Step 

Who? 

What  happens? 

1 

Team 

member 

Scans  the  artifact  and  tabs,  highlights,  or 
annotates  areas  that  are  related  to  M  &  A.5 

2 

Team 

member 

Reviews  relevant  M  &  A  areas  in  light  of  MAID 
criteria  that  are  relevant  to  the  particular  artifact. 

3. 

Team 

member 

Rates  relevant  MAID  criteria.  For  each  rating,  a 
reference  to  the  artifact  section  is  documented  as 
well  as  the  rationale  for  the  rating. 

3 

Team 

lead 

Assesses  progress  of  evaluation  and  facilitates 
communication  among  team  members  and 
between  the  team  and  the  client. 

Table  continues  on  next  page 


5  In  some  cases,  M  &  A  information  is  only  a  part  or  section  of  an  artifact.  Experience  has  shown  that 
an  initial  scan  of  the  document  to  locate  these  areas  is  better  than  a  detailed  page-by-page  review. 
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Procedure, 

cont. 


The  MAID  criteria 
form 


Step 

Who? 

What  happens? 

4 

Team  lead, 

Team  members, 
Client  POC, 

Client  SME(s) 

Team  lead  sets  up  meeting  with  client 

POC  and  client  SMEs  on  an  as-needed 

basis  to  address  need  for  clarification  or 

need  for  access  to  other  information. 

5 

Team  lead 

Compiles  and  organizes  the  team 
member  results.  Ratings  and 
accompanying  information  are 
consolidated  into  one  file  to  provide  a 
single  view  of  the  results. 

A  spreadsheet  can  be  used  to  capture  the  results  of  the  evaluation.  A 
separate  worksheet  can  be  used  to  record  information  for  each  MAID 
category  (i.e.,  for  M  &  A  Planning,  Data  Collection  &  Storage,  Data 
Analysis,  M  &  A  Reporting,  and  M  &  A  -  General). 

The  table  below  describes  the  information  that  should  be  captured 
when  rating  the  satisfaction  of  each  criterion. 


Item 

Description 

Criterion 

number 

The  number  assigned  to  the  criterion  in  the 
document,  Measurement  and  Analysis 

Infrastructure  Diagnostic  (MAID)  Evaluation 

Criteria,  Version  1.0  [SEMA  2009]. 

Criterion 

The  evaluation  criterion  description. 

Rating 

The  evaluator’s  rating  of  how  well  the  criterion  is 
satisfied. 

Evidence  for 
rating 

A  pointer  to  the  location  within  the  artifact  that  is 
being  examined  that  supports  the  criterion  rating. 

Rationale 

The  reasoning  used  by  the  evaluator  to  assign  the 
rating.  This  field  is  optional.  It  is  used  in  cases 
where  the  reason  is  not  obvious  and  additional 
explanation  is  necessary. 

Table  continues  on  next  page 
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The  MAID  criteria 
form,  cont. 


Rating  scale 


Product 


Item 

Description 

Interview 

question(s) 

This  field  is  reserved  for  documenting  interview 
questions  that  will  be  used  during  Phase  3:  On-site 
Evaluation.  The  interview  questions  are  typically 
intended  to  clarify  criterion  issues  that  are  not 
elaborated  clearly  by  the  artifact  being  examined. 

Notes 

The  notes  field  is  intended  to  capture  any  other 
information  that  the  evaluator  wants  to  record 
about  the  criterion  being  investigated. 

A  rating  is  used  to  capture  the  evaluation  of  a  criterion.  Each  criterion 
is  rated  using  one  and  only  one  of  the  ratings  listed  below. 

Rating 

Description 

Very  adequate 

Exceeds  the  standard  for  this  criterion  or  is  viewed 
as  a  superior  implementation. 

Adequate 

Satisfies  the  criterion  sufficiently. 

Slightly 

inadequate 

Criterion  is  almost  satisfied.  By  making  a  slight 
change  to  the  condition  or  practice  addressed  by 
the  criterion,  a  rating  of  adequate  could  be 
achieved. 

Very 

inadequate 

The  condition  or  practice  is  addressed  by  the 
organization  but  in  an  inadequate  way. 

Completely 

missing 

The  condition  or  practice  does  not  exist  based  on 
examination  of  the  M  &  A  artifact. 

Doesn’t  apply 

The  particular  criterion  does  not  apply  to  the 
organizational  context. 

MAID  criteria  form  -  Phase  2  results  that  includes 

•  MAID  criteria  ratings 

•  evidence  for  ratings 

•  rationale  for  ratings 

•  potential  interview  questions 


35  |  CMU/SEI-2010-TR-035 


Additional 

guidance 


•  It  is  helpful  when  documents  are  provided  in  a  format  that  can  be 
edited  by  the  evaluator  (for  example,  to  allow  the  addition  of 
comments,  annotations,  and  bookmarks).  However  some  M  &  A 
artifacts  may  be  provided  in  portable  document  format  (PDF)  or  in 
a  graphic  format  language  (such  as  JPEG).  In  these  cases,  the 
MAID  evaluator  may  want  to  invest  in  applications  that  provide 
the  capability  to  annotate  such  artifacts.  Using  this  approach, 
digital  copies  of  comments  or  annotations  can  be  preserved. 

•  A  spreadsheet  can  be  used  to  capture  the  results  of  the 
evaluation  and  automatically  tally  the  criteria  ratings. 

•  While  reviewing  and  evaluating  an  artifact,  other  questions  and 
issues  not  directly  related  to  a  specific  criterion  may  emerge. 
These  should  be  recorded  in  a  separate  document  template. 

•  Lessons  learned  regarding  the  MAID  method  should  also  be 
recorded  in  a  separate  document  template.  These  lessons  should 
be  compiled  by  the  team  lead  and  shared.  They  should  also  be 
considered  as  improvement  opportunities  for  future  MAID  efforts. 

•  During  Phase  2:  Artifact  Evaluation,  the  team  leader  should  check 
in  with  other  members  to  assess  progress  and  determine  if  the 
evaluation  is  on  track.  Members  should  share  their  intermediate 
work  with  the  team  leader  as  the  evaluation  is  being  carried  out. 
The  team  leader  can  review  the  work-in-progress  and  provide 
feedback  (e.g.,  as  to  whether  rationale  is  being  documented 
sufficiently). 

•  In  some  cases,  there  will  be  questions  about  an  M  &  A  artifact 
that  require  resolution  in  order  to  continue  the  evaluation  in  an 
efficient  and  effective  way.  For  example,  parochial  terminology  or 
undefined  organizational  acronyms  might  be  used  in  an  artifact. 

Instances  such  as  these  should  be  noted  and  a  meeting  or  some 
other  form  of  communication  should  be  scheduled  with  the  client 
POC  by  the  team  lead. 

SMEs  may  be  requested  to  participate  in  a  meeting  or 
teleconference  to  clarify  an  issue  so  the  evaluation  can  continue. 

•  Team  members  should  meet  during  the  evaluation  to  discuss 
their  experience  and  any  lessons  learned  that  could  help  other 
members  as  they  continue  their  evaluation. 
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2.3  Perform  Quality  Audit  of  Results 


Activity  During  this  activity,  the  team  lead  conducts  a  thorough  review  of  the 

description  compiled  results. 


Who 

participates? 


Team  lead 
Team  members 


Information  MAID  criteria  form  -  Phase  2  results  [2.2] 

Needed 


Procedure 


The  table  below  describes  the  key  tasks  of  this  activity. 


Step 

Who? 

What  happens? 

1 

Team  lead 

Reviews  the  consolidated  results  and 

checks  for  the  following: 

•  all  criteria  have  been  rated6 

•  evidence  is  provided  for  all  ratings 
when  appropriate 

•  rationale  for  each  rating  is  provided  if 
evaluator’s  reasoning  is  not  self- 
evident 

•  candidate  interview  questions  (for 

Phase  3:  On-site  Evaluation)  are  clear 
and  understandable 

2 

Team  lead, 

Team  member 

Resolve  any  issues  that  the  team  lead 
could  not  resolve  during  step  1. 

Products  Quality-audited  MAID  criteria  form  -  Phase  2  results 


6  That  is,  all  MAID  criteria  which  are  included  within  the  scope. 
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2.4  Prepare  for  On-Site  Evaluation 


Overview 


Activities  during 
this  stage 


During  this  activity,  preparations  are  made  for  conducting  the  on-site 
evaluation.  Informational  materials  are  prepared  to  support  interview 
sessions  and  a  detailed  agenda  is  coordinated  for  the  on-site 
evaluation. 


This  stage  comprises  the  following  activities: 

2.4.1  Prepare  On-site  Evaluation  Agenda 

2.4.2  Prepare  Materials 

2.4.3  Administer  MAID  Questionnaire 

2.4.4  Manage  Logistics 
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2.4.1  Prepare  On-Site  Evaluation  Agenda 


Activity 

description 

Who 

participates? 


Information 

needed 


Procedure 


During  this  activity,  the  team  lead  works  closely  with  the  client  POC  to 
put  together  a  detailed  plan  for  the  on-site  evaluation. 


•  Team  lead 

•  Client  POC 


•  M  &  A  artifact  inventory  [1 .3.2] 

•  Approved  MAID  plan  [1.3.5] 

•  Quality-audited  MAID  criteria  form  -  Phase  2  results  [2.3] 


The  table  below  describes  the  key  tasks  of  this  activity. 


Step 

Who? 

What  happens? 

1 

Team  lead 

Using  the  M  &  A  artifact  inventory  and  the 
Quality-audited  Phase  2  artifact  evaluation 
results,  creates  a  separate  document  and 
organizes  the  list  of  interview  questions 
according  to  the  client  SME  responsible  for 
the  artifact. 

2 

Team  lead 

Estimates  the  amount  of  time  required  to 
interview  each  client  SME  based  on  the 
results  of  step  1  and  the  list  in  the  draft 
agenda. 

3 

Team  lead 

Reviews  the  Approved  MAID  plan,  identifies 

other  evaluation  activities  that  are  included  in 
the  plan,  estimates  the  duration,  and  adds  to 
the  draft  agenda. 

4 

Team  lead 

Identifies  any  other  needed  activities  (e.g., 

MAID  orientation  presentation,  MAID  team 
prep  and  analysis  sessions,  outbrief 
presentation  preparation,  outbrief  reporting), 
estimates  duration  of  these  activities  and  adds 
to  the  draft  agenda. 
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Procedure,  cont. 


The  table  below  describes  the  key  tasks  of  this  activity. 


Products 


Additional 

guidance 


Step 

Who? 

What  happens? 

5 

Team  lead 

Organizes  the  information  from  steps  1-4 
into  a  draft  on-site  agenda  that  includes  the 
following: 

•  activity 

•  who  participates  in  activity 

•  start  and  stop  time 

•  activity  duration 

•  brief  activity  description 

6 

Team  lead, 

Client  POC 

Team  lead  reviews  the  draft  agenda  with 
the  client  POC  and  makes  any  necessary 
adjustments. 

7 

Client  POC, 

Client  SMEs, 

Stakeholders, 

Sponsor 

Client  POC  reviews  the  draft  agenda  with 
all  individuals  who  participate  in  any  activity 
identified  on  the  agenda.  Change  requests 
are  recorded  by  the  client  POC. 

8 

Client  POC, 

Team  lead 

Client  POC  communicates  change 
requests  to  the  team  lead. 

9 

Team  lead 

Revises  the  agenda  and  reviews  the 
changes  with  the  client  POC  who  then 
approves  the  agenda. 

Approved  agenda  for  on-site  evaluation 


See  Appendix  B  on  page  69  for  an  example  of  an  Approved  agenda 
for  on-site  evaluation. 
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2.4.2  Prepare  Materials 


Activity 

description 

Who 

participates? 


Information 

needed 


Procedure 


During  this  activity,  materials  that  support  the  on-site  evaluation  are 
prepared. 


•  Team  lead 

•  Team  members 


•  M  &  A  artifact  inventory  [1 .3.2] 

•  Approved  MAID  plan  [1.3.5] 

•  Approved  agenda  for  on-site  evaluation  [2.4.1  ] 

•  Quality-audited  MAID  criteria  form  -  Phase  2  results  [2.3] 


The  table  below  describes  the  key  tasks  of  this  activity. 


Step 

Who? 

What  happens? 

1 

Team  lead 

Organizes  a  list  of  questions  for  each  client 

SME  that  will  be  interviewed.  Part  of  this  step 
was  performed  during  Activity  2.4.1  to 
estimate  the  duration  of  an  interview.  Here, 
the  questions  are  refined  and  sequenced  into 
an  interview  script. 

2 

Client  POC 

Develops  a  list  of  individual  stakeholders  that 
will  be  surveyed  about  M  &  A  practices  in  their 
organizations.  If  survey  is  Internet-based,  then 
email  addresses  for  each  individual  are 
provided. 

3 

Team  lead 

Develops  a  self-administered  questionnaire  to 
solicit  additional  input  from  all  stakeholders  in 
the  M  &  A  system. 

4 

Team  lead 

Develops  an  orientation  briefing  that  will  be 
used  at  the  start  of  the  Phase  3:  On-Site 

Evaluation. 
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Products 


•  Interview  questions  organized  by  interviewee 

•  Contact  list  for  questionnaire  target  audience 

•  Self-administered  questionnaire 

•  Orientation  presentation  from  Phase  3:  On-site  Evaluation 


Additional 

guidance 


Interview  questions  organized  by  interviewee 

Information  gleaned  from  the  survey  results  may  provide 
motivation  for  additional  probing  questions  about  the 
organization’s  M  &  A  practices.  For  example,  respondents  might 
suggest  that  actual  practices  do  not  always  adhere  to  the 
documented  M  &  A  practices.  Based  on  such  information,  the 
MAID  evaluation  team  may  decide  to  validate  such  assertions 
during  the  on-site  evaluation. 

Many  interview  questions  from  the  product  Quality-audited  Phase 
2  artifact  evaluation  results  will  be  used  to  develop  the  interview 
questions.  However,  additional  questions  are  added  that  ask 
about  compliance  to  the  documented  M  &  A  plans  and 
procedures. 

Contact  list  for  questionnaire  target  audience 

In  many  cases,  it  is  impractical  to  interview  every  stakeholder  in 
the  organization’s  M  &  A  program.  A  survey  can  be  used  to  obtain 
opinions  about  the  M  &  A  program  and  provide  an  additional 
perspective  on  implementation  issues. 

The  contact  list  will  include  email  addresses  when  the 
questionnaire  is  to  be  administered  over  the  Internet.  There  are 
many  easy-to-use  web-based  products  that  can  be  used  to 
implement  the  survey. 

Self-administered  questionnaire 

The  questionnaire  will  solicit  feedback  on  the  implementation  of 
the  M  &  A  program.  It  can  be  distributed  to  everyone  who  is 
associated  with  the  M  &  A  program. 

The  MAID  questionnaire  can  be  distributed  as  a  paper  form  or  as 
a  web-based  instrument.  A  web-based  instrument  has  many 
advantages,  including  ease  of  distribution  and  automated  support 
for  tracking  and  data  collection. 

For  additional  guidance  on  developing  a  survey,  see  the  SEI 
handbook  Designing  Effective  Surveys  [Kasunic  2005]. 

Questionnaire  item  examples  are  provided  in  Appendix  C. 
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Additional 
guidance,  cont. 


Orientation  presentation  from  Phase  3:  On-site  Evaluation 

The  orientation  provides  a  way  to  help  ensure  that  all 
stakeholders  have  a  shared  understanding  of  what  MAID  is  and 
not.  At  a  minimum,  the  presentation  covers 

o  the  objectives  and  benefits  of  the  MAID  method 

o  a  summary  of  the  MAID  method  by  phase 

o  a  review  of  the  agenda  for  the  duration  of  the  on-site 
evaluation 

o  a  summary  of  what  happens  during  the  interview 
sessions 

o  Q  &  A 

Generally  speaking,  the  presentation  should  be  tailored  to  the 
information  needs  of  the  audience.  It  should  also  address  any 
issues  or  concerns  that  the  audience  might  have. 

The  MAID  team  leader  should  review  the  presentation  with  the 
client  POC  to  test  whether  the  presentation  is  on  target  for  the 
audience. 
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2.4.3  Administer  MAID  Questionnaire 


Activity 

description 


Who 

participates? 

Information 

needed 


Procedure 


The  MAID  questionnaire  is  an  instrument  that  provides  an  additional 
way  of  collecting  information  about  the  organization’s  M  &  A 
practices.  The  questionnaire  is  a  valuable  input  as  it  can  be  used  to 
obtain  feedback  from  all  staff  members  who  are  impacted  by  the 
organization’s  M  &  A  program. 


•  Team  lead 

•  Questionnaire  respondents 


•  Contact  list  for  questionnaire  target  audience  [2.4.2] 

•  Self-administered  questionnaire  [2.4.2] 

•  Interview  questions  organized  by  interviewee  [2.4.2] 


The  table  below  describes  the  key  tasks  of  this  activity. 


Step 

Who? 

What  happens? 

1 

MAID  Lead 

Using  the  Contact  list  for  questionnaire  target 
audience  as  a  source,  an  invitation  to 
participate  in  the  MAID  survey  is  sent  to 
questionnaire  respondents. 

2 

Questionnaire 

respondents 

Complete  the  questionnaire. 

3 

MAID  Lead 

Analyzes  survey  results. 

4 

MAID  Lead 

Based  on  results,  determines  if  probing 
questions  should  be  added  to  the  Interview 
questions  organized  by  interviewees ;  adds 
questions  if  appropriate. 

5 

MAID  Lead 

Formats  the  questionnaire  results  for 
reporting  purposes  (i.e.,  creates  graphical 
rendering). 
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Additional 

guidance 


Products 


•  The  questionnaire  results  are  rolled  into  the  overall  MAID  results 
and  reported  to  the  sponsor  during  Phase  4:  Report  Results. 
However,  the  results  can  be  prepared  at  this  time  by  developing 
illustrative  graphs  and  tables  that  can  be  inserted  into  the  final 
report. 


•  Questionnaire  results 

•  Revised  interview  questions  organized  by  interviewee  (if 
applicable) 

•  Questionnaire  results  organized  for  reporting  purposes 
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2.4.4  Manage  Logistics 


Activity 

description 


Who 

participates? 


Information 

needed 


Procedure 


Products 


During  this  activity,  the  logistical  details  of  the  on-site  evaluation  are 
negotiated  and  documented.  Checklists  and  action  item  tracking 
mechanisms  are  effective  ways  to  manage  and  track  these  tasks. 


•  Team  lead 

•  Client  POC 


•  Approved  MAID  plan  [1 .3.5] 

•  Approved  agenda  for  on-site  evaluation  [2.4.1  ] 


The  table  below  describes  the  key  tasks  of  this  activity. 


Step  Who?  What  happens? 

1 

Team  lead 

Develops  an  action  item  list  to  plan  and 
document  logistical  tasking. 

2 

Team  lead, 

Client  POC 

Plan  the  details  of  the  on-site  evaluation  and 

add  tasks  to  the  action  item  list. 

3 

Team  lead 

Updates  the  action  item  list  as  needed. 

Action  item  list  (to  support  Phase  3:  On-site  Evaluation) 
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Additional 

guidance 


The  action  item  list  template  should  include  columns  for  the  following 
information: 

•  task  description 

•  task  responsibility 

•  task  due  date 

•  task  completion  date 

The  person  managing  the  action  item  list  should  ensure  the  following: 

•  the  list  is  shared  with  all  individuals  who  are  responsible  for  a 
task  that  appears  in  the  list 

•  version  control  is  maintained 

•  a  new  version  is  emailed  to  affected  individuals  whenever 
there  is  an  addition  to  the  list,  or  the  list  should  be  maintained 
in  a  shared  repository 

Examples  of  actions  needed  to  support  Phase  3:  On-site  Evaluation: 

•  communicating  with  individuals  who  will  be  interviewed  during 
this  phase  (e.g.,  a  “what-to-expect”  email) 

•  transportation  to  and  from  the  site 

•  identifying  hotels  for  individuals  traveling  to  the  site 

•  ordering  meals  or  identifying  nearby  restaurants 

•  arranging  for  a  conference  room  or  other  appropriate  space  to 
conduct  the  on-site  orientation  meeting 

•  arranging  for  a  conference  room  or  other  appropriate  space  to 
conduct  interviews 

•  arranging  private  work  space  for  the  MAID  team  to  conduct 
off-line  work 

•  meeting  security  requirements  at  the  site  (e.g.,  getting  visitor 
badges) 

•  obtaining  access  to  databases  that  are  part  of  the  MAID 
scope 

•  arranging  access  to  equipment  such  as  printers  and  copiers 
and  other  supplies 
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3  On-Site  Evaluation 


1.  Collaborative 
planning 


Introduction  During  Phase  3:  On-Site  Evaluation,  the  MAID  team  conducts  a 

number  of  interviews  and  examinations  at  the  organization’s  site. 
These  include 

•  interviews  with  client  SMEs7 

•  interviews  with  stakeholders 

•  examination  of  M  &  A  data  repositories 

•  examination  of  any  M  &  A  artifact  that  could  not  be  evaluated 
off-site 


Before  you  All  required  actions  recorded  in  the  action  item  list  need  to  be 

begin  sufficiently  addressed  (see  Activity  2.4.3)  before  beginning  this 

phase. 

Stages  The  stages  included  in  this  phase  are 

3.1  Conduct  Orientation  Meeting 

3.2  Conduct  Examinations  of  M  &  A  Data  Repositories 

3.3  Conduct  Interviews 


7  In  some  cases,  the  sponsor  may  also  be  an  SME. 
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3.1  Conduct  Orientation  Meeting 


Activity 

description 


Who 

participates? 


Information 

needed 


Procedure 


An  orientation  meeting  is  the  first  activity  of  the  on-site  evaluation. 
The  purpose  of  the  meeting  is  to 

•  provide  an  overview  of  MAID,  including  objectives  and 
benefits 

•  describe  what  will  happen  during  the  on-site  evaluation 

•  answer  questions  from  the  audience 


•  Team  lead 

•  Team  members 
.  Client  POC 

•  Sponsor 

•  Client  SMEs 

•  Stakeholders 


•  Approved  agenda  for  on-site  evaluation  [2.4.1  ] 

•  Orientation  presentation  from  Phase  3:  On-site  Evaluation  [2.4.2] 


The  table  below  describes  the  key  tasks  of  this  activity. 


Step  Who?  What  happens? 

1 

Client  POC 

Before  the  meeting, 

•  prepares  conference  room  for  the  meeting 

•  ensures  that  a  projection  system  is  in 
operating  order 

2 

Sponsor 

Kicks  off  the  meeting  by  emphasizing  support 
for  this  effort  and  the  benefits  that  will  be 
derived  from  the  MAID  evaluation  (sponsor 
perspective). 

3 

Client  POC 

Introduces  the  MAID  team. 

Table  continues  on  next  page 
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Procedure,  cont. 


Products 


Step  Who? 

What  happens? 

4 

Team  lead 

•  Presents  brief  MAID  overview  to  provide 

context 

•  Presents  the  agenda  for  the  on-site 
evaluation 

•  Describes  what  has  already  happened 
(i.e.,  during  Phase  1:  Collaborative 

Planning  and  Phase  2:  Artifact  Evaluation) 

•  Describes  the  MAID  questionnaire  (if 
applicable) 

•  Describes  what  will  happen  during  the 
interviews 

•  Describes  what  will  happen  after  the  on¬ 
site  evaluation 

•  Conducts  question  and  answer  session 

1-4 

Team 

member 

Records  issues,  action  items,  or  any 
observations  to  support  post-MAID  lessons 
learned. 

Orientation  meeting  record 
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3.2  Conduct  Examinations  of  M  &  A  Data  Repositories 


Activity  During  this  activity,  MAID  team  members  examine  M  &  A  data 

description  repositories  that  are  included  in  the  MAID  scope. 

A  number  of  MAID  evaluation  criteria  address  data  quality  and 
storage  mechanisms.  These  criteria  are  rated  as  an  outcome  of  the 
evaluation.  Also,  other  MAID  criteria  address  procedural  aspects  of 
how  data  is  stored  and  managed.  During  this  examination,  the  MAID 
team  members  validate  that  the  implementation  follows  the 
documented  procedures. 


Who 

participates? 


Team  member(s)8 
Client  SME 


Information 

needed 


Quality-audited  Phase  2  artifact  evaluation  results  [2.3] 
Client  M  &  A  repository 


Procedure 


The  table  below  describes  the  key  tasks  of  this  activity. 


Step  Who?  What  happens? 

1 

Client  POC  or 

Client  SME 

Provides  access  for  team  members  to  the 
client  data  repository  to  be  examined. 

2 

Team 

member(s) 

Examines  the  M  &  A  data  repository  and 
rates  the  applicable  MAID  criteria. 

Products 


Intermediate  Phase  2-3  artifact  evaluation  results 


Additional 

guidance 


•  Criteria  listed  in  the  MAID  criteria  category  “Data  Collection  & 
Storage”  will  be  particularly  relevant  to  this  examination. 

•  This  examination  provides  an  opportunity  to  investigate  whether 
implementation  follows  M  &  A  documented  plans  and  procedures. 


8  The  team  lead  could  also  fill  the  role  of  team  member. 
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3.3  Conduct  Interviews 


Activity  During  this  activity,  client  SMEs  are  interviewed  about  the  M  &  A 

description  practices  of  their  organization. 


Who 

participates? 


Team  member(s)9 
Client  SMEs 


Information 

needed 


Interview  questions  organized  by  interviewee  [2.4.2] 
Intermediate  Phase  2-3  artifact  evaluation  results  [3.2] 


Procedure 


The  table  below  describes  the  key  tasks  of  this  activity. 


Step  Who?  What  happens? 

1 

Team 

members 

Before  the  interview,  the  team  members 
review  the  questions  to  prepare  for  the 
interview. 

2 

Team 

members, 

Client  SMEs 

During  the  interview,  one  team  member  acts 
as  primary  interviewer  while  other  team 
member(s)  record  the  interview  responses. 

3 

Team 

members 

After  the  interview,  the  team  members  review 
the  interview  transcript  for  accuracy  and 
completeness  and  revise  transcript  if 
necessary. 

Products  Interview  transcripts 


9  The  team  lead  could  also  fill  the  role  of  team  member. 
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Additional 

guidance 


•  At  least  two  team  members  are  needed  for  each  interview.  One 
member  moderates  the  interview  and  the  other  member  records 
interviewee  responses. 

•  The  product  Intermediate  Phase  2-3  artifact  evaluation  results  is 
available  as  a  reference  during  the  interviews  in  the  event  that 
team  members  want  to  refer  to  information  in  it. 

•  Further  guidance  for  conducting  interviews  is  provided  in 
Appendix  D. 
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4  Report  Results 


1.  Collaborative 

2.  Artifact 

3.  On-site 

4.  Report 

planning 

evaluation 

evaluation 

results 

Introduction  During  Phase  4:  Report  Results,  the  MAID  team  compiles  the  results 

from  Phases  2  and  3  to  generate  a  report  that  is  delivered  to  the 
client.  The  report  can  be  mediated  in  a  number  of  ways. 


Stages  The  stages  included  in  this  phase  are: 

4.1  Analyze  On-site  Evaluation  Results 

4.2  Derive  Key  Findings 

4.3  Deliver  Key  Findings 

4.4  Plan  Next  Steps 
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4.1  Analyze  On-Site  Evaluation  Results 


Activity 

description 


During  this  activity,  MAID  team  members  compile  and  analyze  the 
results  from  the  on-site  examinations,  including 

•  interview  transcripts 

•  M  &  A  database  examinations 

•  any  other  M  &  A  artifacts  that  were  evaluated  during 


Phase  2:  Artifact  Evaluation 


10 


Who 

participates? 


Team  lead 
Team  members 


Information 

needed 


Intermediate  Phase  2-3  artifact  evaluation  results  [3.2] 
Interview  transcripts  [3.3] 


Procedure  The  table  below  describes  the  key  tasks  of  this  activity. 


Step  Who?  What  happens? 

1 

Team  lead, 

Team  member(s) 

Review  interview  transcripts  and  update 
the  Intermediate  Phase  2-3  artifact 

evaluation  results. 

2 

Team  lead, 

Team  member(s) 

If  there  are  any  other  artifacts  that  have 
been  examined,  then  the  Intermediate 

Phase  2-3  artifact  evaluation  results  are 
updated. 

Products  Final  MAID  criteria  evaluation  results 


10  This  may  include  planning  and  process/procedure  documentation  that  was  not  included  as  part  of 
the  Phase  2  evaluations. 
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4.2  Derive  Key  Findings 


Activity 

description 


Who 

participates? 


Information 

needed 


Procedure 


During  this  activity,  MAID  team  members  prepare  the  results  for 
reporting  to  the  client  organization.  The  evaluation  results  are 
reviewed  in  light  of  the  MAID  objectives  and  the  MAID  plan.  The 
results  are  translated  into  key  findings  that  summarize  strengths  and 
weaknesses.  The  Final  MAID  criteria  evaluation  results  can  be 
included  as  an  appendix  to  the  report. 


•  Team  lead 

•  Team  members 


•  List  of  client  SMEs  [1 .2] 

•  Approved  MAID  plan  [1 .3.5] 

•  Questionnaire  results  organized  for  reporting  purposes  [2.4.3] 

•  Final  MAID  criteria  evaluation  results  [4.1] 


The  table  below  describes  the  key  tasks  of  this  activity. 


Step  Who?  What  happens? 

1 

Team  lead, 

Team  member(s) 

Review  the  list  of  outputs  with  descriptions 
that  will  be  developed  based  on  the 
results  of  the  MAID  scope  that  was 
included  in  the  Approved  MAID  plan. 

2 

Team  lead, 

Team  member(s) 

Develop  an  annotated  outline  of  the  MAID 
report. 

3 

Team  lead, 

Team  member(s) 

Validate  that  the  outline  developed  in  step 

2  addresses  the  List  of  MAID  objectives 
that  should  be  included  in  the  plan. 

Continued  on  next  page 
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Procedure, 

Continued 


Products 


Step  Who?  What  happens? 

4 

Team  lead,  Team 

members 

Using  the  Final  MAID  criteria 
evaluation  results  as  a  reference, 
prepare  a  list  of  findings  that  are 
representative  of  the  results  in  each 
MAID  criteria  category  that  was 
included  in  the  scope. 

5 

Team  lead 

Publishes  the  Final  MAID  criteria 
evaluation  findings. 

Final  MAID  criteria  evaluation  findings 
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4.3  Deliver  Key  Findings 


Activity 

description 


Who 

participates? 


Information 

needed 


The  key  findings  contain  a  summary  of  the  strengths  and  weaknesses 
of  the  organization’s  M  &  A  program.  These  findings  can  be 
accompanied  by  the  detailed  evaluation  results  that  were  used  to 
generate  the  findings. 

The  key  findings  should  be  delivered  in  a  way  that  supports  decision 
making  and  future  M  &  A  planning. 

How  the  key  findings  are  communicated  is  decided  during  Phase  1 : 
Collaborative  Planning.  The  findings  can  be  delivered  in  the  following 
formats: 

•  an  executive  summary  of  key  findings 

•  a  detailed  slide  presentation 

•  a  detailed  textual  report 

•  a  combination  of  the  above 


•  Team  lead 

•  Team  members  (optional) 

•  Sponsor 

•  Client  POC 

•  Client  SMEs 

•  Questionnaire  respondents 

•  Stakeholders 

Final  MAID  criteria  evaluation  findings  [4.2] 
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Procedure 


The  table  below  describes  the  key  tasks  of  this  activity. 


Products 


Additional 

guidance 


Step  Who?  What  happens? 

1 

Team  lead,  Client 

POC 

Team  lead  and  client  POC  negotiate  a 
list  of  potential  dates  and  times  for 
delivery  of  the  report. 

2 

Client  POC, 

Client  SMEs, 

Questionnaire 

respondents, 

Stakeholders 

If  an  oral  presentation  of  the  results 
(supported  by  slides)  is  to  be 
delivered,  the  client  POC  coordinates 
a  date/time  that  optimizes  attendance 
by  organization  members. 

3 

Team  lead 

Delivers  the  MAID  final  report. 

Delivered  MAID  results 


When  presenting  results,  it  is  important  that  information  is  not 
attributable  to  any  single  individual  within  the  organization.  The 
commitment  of  non-attribution  during  the  interviewing  process  must 
be  upheld. 
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4.4  Plan  Next  Steps 


Activity 

description 


Who 

participates? 


Information 

needed 


Procedure 


Products 


Additional 

guidance 


Planning  next  steps  is  an  optional  activity  but  is  recommended. 
Follow-on  activities  may  include 

•  delivery  of  a  detailed  report,  if  it  was  not  originally  included  in 
the  Approved  MAID  plan  [1 .3.5] 

•  generation  of  an  action  plan  to  address  the  MAID  findings 


•  Sponsor 

•  Stakeholders 

•  Team  lead 


Delivered  MAID  results  [4.3] 


The  table  below  describes  the  key  tasks  of  this  activity. 


Step  Who?  What  happens? 

1 

Sponsor,  Team 
lead 

Negotiate  the  role  that  MAID  team  lead 
will  play  during  this  activity. 

2 

Sponsor, 

Stakeholders, 

Team  lead 
(optional) 

Identify  specific  next  steps  to  address  the 
findings  reported  in  the  Delivered  MAID 
results. 

Action  plan 


•  In  addition  to  specifying  the  follow-on  activities  that  are  to  be 
performed,  the  action  plan  typically  includes  the  assignment  of 
responsibility  for  performing  each  activity,  a  schedule,  and 
estimated  resources. 

•  One  outcome  of  this  activity  might  be  the  development  of  a 
detailed  MAID  final  report  with  recommendations,  if  this  is 
something  that  was  not  originally  contracted  for  during  Phase  1 : 
Collaborative  Planning. 
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Appendix  A:  MAID  Summary 

Sec. 

Title 

Information  Needed 

Products 

1. 

Collaborative  Planning 

1.1 

Establish  Scope 

1.1.1 

Determine  Maid  Objectives 

• 

Description  of  the  organization’s  business 
goals  or  business  needs 

List  of  MAID  objectives 

1.1.2 

Determine  Constraints 

• 

• 

List  of  MAID  objectives  [1.1.1] 

Estimates  of  personnel  availability 

•  List  of  constraints  to  be  met  (accompanied  by 
rationale  for  decisions  made) 

•  List  of  personnel  availability 

•  List  of  revised  MAID  objectives  (based  on  the 
results  of  this  activity) 

1.1.3 

Determine  MAID  Scope 

• 

• 

List  of  revised  MAID  objectives  [1.1.2] 

List  of  constraints  to  be  met  [1.1.2] 

•  MAID  scope  statement  that  includes: 

-  identification  of  organizational  entities  that 
will  be  included  in  the  MAID  evaluation 

-  identification  of  MAID  criteria  categories 
that  will  be  used  for  each  organizational 
entity 

-  rationale  for  decisions  made 
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Sec. 

Title 

Information  Needed 

Products 

1.2 

Establish  Roles  and 
Expectations 

•  List  of  MAID  team  members  with  contact 
information  and  availability  information 

•  List  of  client  SMEs  that  includes  the  following 
descriptors 

-  name 

-  contact  information  (phone,  email) 

-  organization/project 

-  organization/project  title 

M  &  A  role  in  organization 

M  &  A  SME  area  designation  -  within 

MAID  scope 

-  availability  information 

1.3 

Develop  Plan  and  Schedule 

1.3.1 

Determine  MAID  Outputs 

• 

• 

List  of  revised  MAID  objectives  [1.1.2] 

Input  from  Sponsor  to  client  POC 

List  of  outputs  with  descriptions  that  will  be 
developed  based  on  the  results  of  the  MAID 

1.3.2 

Determine  MAID  Inputs 

• 

List  of  client  SMEs  [1 .2] 

M  &  A  artifact  inventory 

1.3.3 

Tailor  Method 

• 

• 

• 

List  of  revised  MAID  objectives  [1.1.2] 

List  of  constraints  to  be  met  accompanied 
by  rationale  for  decisions  made  [1 .1 .2] 

MAID  scope  statement  [1 .1 .3] 

Approved  tailoring  decisions 
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Sec.  Title 

Information  Needed 

Products 

1.3.4 

Determine  Cost  and 

Schedule 

• 

• 

• 

List  of  revised  MAID  objectives  [1.1.2] 

MAID  scope  statement  [1 .1 .3] 

Approved  tailoring  decisions  [1 .3.3] 

Approved  cost  and  schedule  estimates  for  MAID 
effort 

1.3.5 

Obtain  Commitment  to  the 

Plan 

• 

• 

• 

List  of  MAID  revised  Objectives  [1.1 .2] 

List  of  personnel  availability  [1 .1 .2] 

The  following  parts  of  the  MAID  scope 

Approved  MAID  plan 

statement  [1.1.3]: 

-  Identification  of  organizational  entities 
that  will  be  included  in  the  MAID 
evaluation 

-  Identification  of  MAID  criteria  categories 
that  will  be  used  for  each  organizational 
entity 

•  Approved  cost  and  schedule  estimates  for 
MAID  effort  [1 .3.4] 


2  Artifact  Evaluation 


2.1  Prepare  for  Artifact  Evaluation 


•  List  of  MAID  team  members  with  contact  M  &  A  artifacts  received  from  the  client 
information  and  availability  information  organization 

[1.2] 

•  List  of  client  SMEs  [1.2] 

•  M  &  A  artifact  inventory  [1 .3.2] 

•  Approved  MAID  plan  [1 .3.5] 
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Sec. 

Title 

Information  Needed 

Products 

2.2 

Conduct  M  &  A  Artifact 

• 

M  &  A  artifact  inventory  [1 .3.2] 

MAID  criteria  form  -  Phase  2  results  that  includes 

Evaluation 

• 

M  &  A  artifacts  received  from  the  client 

•  MAID  criteria  ratings 

organization  [2.1] 

•  evidence  for  ratings 

• 

MAID  criteria  form 

•  rationale  for  ratings 

•  potential  interview  questions 

2.3 

Perform  Quality  Audit  of  Results 

MAID  criteria  form  -  Phase  2  results  [2.2] 

Quality-audited  MAID  criteria  form  -  Phase  2 

results 

2.4 

Prepare  for  On-Site  Evaluation 

2.4.1 

Prepare  On-Site  Evaluation 

• 

M  &  A  artifact  inventory  [1 .3.2] 

Approved  agenda  for  on-site  evaluation 

Agenda 

• 

Approved  MAID  plan  [1 .3.5] 

• 

Quality-audited  MAID  criteria  form  - 
Phase  2  results  [2.3] 

2.4.2 

Prepare  Materials 

• 

M  &  A  artifact  inventory  [1 .3.2] 

•  Interview  questions  organized  by  interviewee 

• 

Approved  MAID  plan  [1 .3.5] 

•  Contact  list  of  questionnaire  target  audience 

• 

Approved  agenda  for  on-site  evaluation 

•  Self-administered  questionnaire 

[2.4.1] 

•  Orientation  presentation  from  Phase  3:  On-site 

• 

Quality-audited  MAID  criteria  form  - 
Phase  2  results  [2.3] 

Evaluation 
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Sec. 

Title 

Information  Needed 

Products 

2.4.3 

Administer  MAID 

Questionnaire 

•  Contact  list  for  questionnaire  target 
audience  [2.4.2] 

•  Self-administered  questionnaire  [2.4.2] 

•  Interview  questions  organized  by 
interviewee  [2.4.2] 

•  Questionnaire  results 

•  Revised  interview  questions  organized  by 
interviewee  (if  applicable) 

•  Questionnaire  results  organized  for  reporting 
purposes 

2.4.4 

Manage  Logistics 

•  Approved  MAID  plan  [1 .3.5] 

•  Approved  agenda  for  on-site  evaluation 
[2.4.1] 

Action  item  list  (to  support  Phase  3:  On-site 
Evaluation) 

3 

On-Site  Evaluation 

3.1 

Conduct  Orientation  Meeting 

•  Approved  agenda  for  on-site  evaluation 
[2.4.1] 

•  Orientation  presentation  from  Phase  3: 
On-site  Evaluation  [2.4.2] 

Orientation  meeting  record 

3.2 

Conduct  examination(s)  of  M  & 

A  data  repositories 

•  Quality-audited  Phase  2  artifact 
evaluation  results  [2.3] 

•  Client  M  &  A  repository 

Intermediate  Phase  2-3  artifact  evaluation  results 

3.3 

Conduct  interviews 

•  Interview  questions  organized  by 
interviewee  [2.4.2] 

•  Intermediate  Phase  2-3  artifact  evaluation 
results  [3.2] 

Interview  transcripts 

4  Report  Results 
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Sec. 

Title 

Information  Needed 

Products 

4.1 

Analyze  on-site  evaluation 

• 

Intermediate  Phase  2-3  artifact  evaluation 

Final  MAID  criteria  evaluation  results 

results 

• 

results  [3.2] 

Interview  transcripts  [3.3] 

4.2 

Derive  key  findings 

• 

List  of  client  SMEs  [1 .2] 

Final  MAID  criteria  evaluation  findings 

• 

Approved  MAID  plan  [1 .3.5] 1 1 

• 

Questionnaire  results  organized  for 
reporting  purposes  [2.4] 

• 

Final  MAID  criteria  evaluation  results  [4.1  ] 

4.3 

Deliver  key  findings 

Final  MAID  criteria  evaluation  findings  [4.2] 

Delivered  MAID  results 

4.4 

Plan  next  steps 

Delivered  MAID  results  [4.3] 

Action  plan 

Note  that  the  Approved  MAID  plan  includes  List  of  MAID  objectives  [1 .1.1],  MAID  scope  statement  [1 .1 .3],  List  of  outputs  with  descriptions  that  will  be 
developed  based  on  the  results  of  the  MAID  [1 .3.4]. 
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Appendix  B:  Example  Agenda  for  On-Site  Evaluation 


This  is  an  example  agenda  for  the  first  day  of  Phase  3:  On-Site  Evaluation.  Note  that  this  is 
only  an  example,  and  the  topic  list  and  times  assigned  to  each  topic  will  depend  on  the 
chosen  scope  of  the  MAID. 


Start 

End 

Time 

Topic 

8:00  AM 

9:00 

1:00 

Kick-off  meeting 

9:00 

9:30 

0:30 

Break  &  set  up  for  interviews 

9:30 

10:15 

0:45 

Interview  -  Data  consumers 

10:15 

10:30 

0:15 

Interview  review  &  prep  for  next  interview 

10:30 

10:40 

0:10 

MAID  Team  Break 

10:30 

11:15 

0:45 

Interview  -  Data  producers 

11:15 

11:30 

0:15 

Interview  review  &  prep  for  next  interview 

11:30 

12:15  PM 

0:45 

Interview  -  Data  storage 

12:15 

12:45 

0:30 

Examination  -  Data  storage 

12:45 

1:00 

0:15 

Interview  review  &  prep  for  next  interview 

1:00 

2:00 

1:00 

Lunch  &  Prep  for  afternoon 

2:00 

2:45 

0:45 

Interview  -  Data  Analysts 

2:45 

3:00 

0:15 

Interview  review  &  prep  for  next  interview 

3:00 

3:10 

0:10 

MAID  Team  Break 

3:10 

3:55 

0:45 

Interview  -  M&A  Info  Designers  and  Presenters 

3:55 

4:10 

0:15 

Interview  review  &  prep  for  next  interview 

4:10 

5:00 

0:50 

MAID  Team  -Day  1  Wrap-up 
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Appendix  C:  Example  Questions  for  Self-Administered 
Questionnaire 


The  following  are  some  example  questions  that  might  be  included  in  a  self-administered 
survey  to  obtain  stakeholder  feedback. 

The  results  of  the  survey  can  be  combined  with  other  MAID  results  from  the  artifact 
evaluation  and  the  interviews  to  provide  a  well-rounded  picture  of  the  health  of  M  &  A 
practices  within  the  organization  undergoing  a  MAID  evaluation. 


1 .  Which  of  the  following  best  describes  the  role  you  play  in  your  organization  (Please 
select  one.) 

□  Executive  or  senior  management 

□  Middle  manager 

□  Project  manager 

□  Project  engineer  or  other  technical  staff 

□  Process  or  quality  engineer 

□  Measurement  specialist 

□  Other  (describe  briefly) 

2.  Please  select  the  response  that  best  describes  your  involvement  with  measurement. 
(Select  a  single  response.) 

□  I  am  a  provider  of  measurement-based  information 

□  I  am  a  user  (consumer)  of  measurement-based  information 

□  I  am  both  a  provider  and  user  (consumer)  of  measurement-based  information 

□  Other  (please  specify) 
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3.  Generally  speaking,  I  believe  that  using  measurement-based  data  helps  my  team  to 
perform  better  than  without  using  it. 

□  Strongly  Agree 

□  Agree 

□  Somewhat  Agree 

□  Not  sure 

□  Somewhat  Disagree 

□  Disagree 

□  Strongly  Disagree 

□  N/A 

4.  The  definitions  of  measures  that  are  used  in  my  organization  are  commonly  understood 
and  consistent. 

□  Strongly  Agree 

□  Agree 

□  Somewhat  Agree 

□  Not  sure 

□  Somewhat  Disagree 

□  Disagree 

□  Strongly  Disagree 

□  N/A 

5.  Select  each  method  that  you  use  to  identify,  collect,  and  analyze  your  measurement  data. 
(You  may  select  more  than  one  response.) 

□  I  do  not  use  a  defined  method. 

□  Goal-Driven  Software  Measurement  method 

□  Practical  Systems  and  Software  (PSM)  method 

□  GQM  (Goal-Question-Metric) 

□  CMMI  Measurement  and  Analysis  Process  Area 

□  Personal  Software  Process  (PSP)  &  Team  Software  Process  (TSP) 

□  ISO  1 5939  (Information  Technology  -  Software  Engineering  -  Software 
Measurement  Process 

□  Other  (please  specify) 
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Please  select  the  response  that  best  describes  you  or  your  team.  (Select  a  single  response 
for  each  numbered  item.) 

6.  There  exist  measurable  criteria  for  the  products  and  services  to  which  I  contribute. 


Frequently 

Occasionally 

Rarely 

Never 

1  don’t  know 

N/A 

□ 

□ 

□ 

□ 

□ 

□ 

1  use  measurement  to  understand  the  quality  of  the  products  and/or  services  that  1  work 

on. 

Frequently 

Occasionally 

Rarely 

Never 

1  don’t  know 

N/A 

□ 

□ 

□ 

□ 

□ 

□ 

My  team  follows  a  documented  process  for  collecting  measurement  data. 

Frequently 

Occasionally 

Rarely 

Never 

1  don’t  know 

N/A 

□ 

□ 

□ 

□ 

□ 

□ 

My  team  follows  a  documented  process  for  reporting  measurement  data  to  management 

Frequently 

Occasionally 

Rarely 

Never 

1  don’t  know 

N/A 

□ 

□ 

□ 

□ 

□ 

□ 

73  |  CMU/SEI-2010-TR-035 


In  the  following  question  the  term  "threshold"  indicates  a  target  or  boundary  that  when 
exceeded  is  evidence  that  a  risk  or  problem  exists. 


10.  Corrective  action  is  taken  when  measurement  data  indicate  that  a  threshold  has  been 
exceeded.  (Select  a  single  response.) 


Frequently 

Occasionally 

Rarely 

Never 

1  don’t  know 

N/A 

□ 

□ 

□ 

□ 

□ 

□ 

11.1  understand  the  purposes  for  the  data  I  collect  or  report.  (Select  a  single  response.) 


Frequently 

Occasionally 

Rarely 

Never 

1  don’t  know 

N/A 

□ 

□ 

□ 

□ 

□ 

□ 

12.  How  often  are  qualified,  well-prepared  people  available  to  work  on  measurement  and 
analysis  in  your  organization  when  you  need  them  (i.e.,  people  with  sufficient 
measurement-related  knowledge,  competence,  or  statistical  sophistication).  (Please 
select  one.) 

□  Almost  always  (>  80%) 

□  Frequently  (between  60%  and  80%) 

□  About  half  the  time  (between  40%  and  60%) 

□  Occasionally  (between  20%  and  40%) 

□  Rarely  (<  20%) 
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13.  How  would  you  best  characterize  the  measurement  and  analysis  training  that  is  available 
in  the  organization?  (Please  select  one.) 

□  Excellent 

□  Good 

□  Adequate 

□  Fair 

□  Poor 
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Appendix  D:  Guidance  for  Conducting  Interviews 


Overview 


Parts  of  the 
interview 


This  appendix  provides  guidance  for  conducting  interviews  during 
Phase  3:  Conducting  On-Site  Evaluation. 


The  interview  meeting  is  a  three-part  process,  as  described  in  the 
table  below. 


Interview  part 

What  happens 

Orientation 

• 

Introductions  and  exchange  of 
business  cards. 

• 

Interviewer  describes  how  this  meeting 
fits  into  the  overall  context  of  what 

MAID  is  attempting  to  accomplish. 

• 

Interviewer  describes  how  the 

interview  will  be  conducted  and  tests 
for  understanding. 

• 

Interviewer  reviews  ground  rules  such 
as  non-attribution  of  information  and 
non-sharing  of  raw  data. 

• 

Interviewer  tests  general 
understanding. 

Information-gathering 

The  interviewer  uses  the  questionnaire  to 
guide  the  interview  and  to  record 
responses. 

Closing 

• 

Interviewer  reviews  key  information 
covered  during  the  interview  and 
confirms  its  accuracy  with  the 
respondent. 

• 

Interviewer  reviews  any  issues  and/or 
action  items. 

• 

Interviewee  provides  feedback  on  the 
interview  process. 

• 

Interviewer  expresses  satisfaction  with 
the  interview  and  makes  a  statement  of 
appreciation. 
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Recommendations 
for  orientation 


Interviewer 

guidelines 


Some  recommendations  for  conducting  the  interview  orientation  are 
provide  below. 

•  Obtain  respondent  contact  information  and  record  it  (if  you  do 
not  have  it  already). 

•  If  you  anticipate  that  a  respondent  may  not  understand  the 
context  for  the  interview  and  how  it  fits  with  the  MAID 
evaluation  and  the  M  &  A  improvement  effort,  you  should  be 
prepared  for  a  5-10  minute  presentation  to  cover  these  topics. 
Use  presentation  slides  to  convey  your  story. 

•  Ensure  that  the  respondent  understands  the  objectives  of  the 
interview  and  how  the  information  from  the  interview  will  be 
used. 

•  Explain  the  ground  rules  for  the  interview  and  suggest 
response  guidelines.  Remind  the  respondent  that  the 
information  will  be  used  by  the  MAID  team  and  responses  will 
not  be  shared  in  a  way  to  disclose  the  identity  of  the  individual 
who  provides  the  information. 

•  Explain  what  will  happen  during  the  course  of  the  interview. 

•  Invite  the  respondent  to  ask  questions. 


The  interviewer  needs  to  be  prepared  for  the  primary  task:  being  an 
effective  listener.  Some  common  pitfalls  that  can  threaten  an  effective 
interview  include 

•  permitting  personal  prejudices  or  deep-seated  convictions  to 
overwhelm  or  impair  comprehension  or  interpretation 

•  interjecting  interviewer  comments  that  can  bias  the  response 

•  preparing  to  answer  questions  before  fully  understanding 
them 

•  listening  only  to  what  is  easy  to  understand 

•  allowing  emotionally-laden  words  to  interfere  with  listening 

•  thinking  you  already  know  what  the  respondent  would  say  to  a 
question  based  on  the  respondent’s  prior  responses  [Barker 
1971,  pp.  61-65] 
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Asking  the 
questions 


•  Use  the  questionnaire  carefully,  but  informally. 

The  questionnaire  includes  topic  areas  that  address  important 
issues;  however,  if  interviewers  focus  too  much  on  the  instrument 
and  read  the  questions,  they  may  appear  unprofessional  and 
disinterested. 

Interviewer  should  memorize  the  first  few  questions  so  they  refer 
to  the  instrument  only  occasionally,  using  eye  contact  and  a 
confident  manner  to  set  the  tone  for  the  interview  and  help 
establish  rapport  with  the  respondent. 

•  Ask  every  question. 

Sometimes  interviewers  will  be  tempted  to  omit  a  question 
because  they  think  they  already  know  what  the  respondent  will 
say.  In  those  cases,  where  the  interviewer  feels  that  a  question 
has  already  been  addressed  (through  some  tangential  aspect  of  a 
different  question),  they  should  still  repeat  the  written  question 
saying  something  like  “I  know  you  may  have  already  mentioned 
this,  but ...” 
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Obtaining 

adequate 

responses 


When  the  respondent  gives  a  brief,  cursory  answer,  the  interviewer 
needs  to  elicit  a  more  thoughtful,  thorough  response.  There  may  be 
times  when  the  response  is  vague  and  where  more  clarity  is  needed. 

Here  are  a  number  of  probing  techniques  that  can  be  used: 

•  The  silent  probe 

The  most  effective  way  to  encourage  someone  to  elaborate  is 
to  do  nothing  at  all — just  pause  and  wait.  This  is  referred  to  as 
the  silent  probe.  It  works  (at  least  in  some  cultures)  because 
the  respondent  is  uncomfortable  with  pauses  or  silence.  It 
suggests  to  respondents  that  the  interviewer  is  waiting, 
listening  for  what  they  will  say  next. 

•  Overt  encouragement 

At  times,  interviewers  can  encourage  the  respondent  directly. 
They  should  try  to  do  so  in  a  way  that  does  not  imply  approval 
or  disapproval  of  what  the  respondent  said  (that  could  bias 
their  subsequent  results).  Overt  encouragement  could  be  as 
simple  as  saying  “uh-huh”  or  “okay”  after  the  respondent 
completes  a  thought. 

•  Elaboration 

Interviewers  can  encourage  more  information  by  asking  for 
elaboration.  For  instance,  it  is  appropriate  to  ask  questions 
such  as,  “Would  you  like  to  elaborate  on  that?”  or  “Is  there 
anything  else  you  would  like  to  add?” 

•  Ask  for  clarification 

Sometimes,  interviewers  can  elicit  greater  detail  by  asking  the 
respondent  to  clarify  something  that  was  said  earlier  by 
saying  something  such  as,  “A  minute  ago  you  were  talking 
about  the  experience  you  had  while  collecting  defect  data. 
Could  you  tell  me  more  about  that?” 

•  Repetition 

Interviewers  sometimes  need  to  verify  the  correctness  of  their 
understanding  of  the  respondent’s  comments.  For  instance, 
suppose  the  respondent  just  described  a  situation  that  was 
problematic  for  him  or  her.  The  interviewer  might  say  “What 
I'm  hearing  you  say  is  that  you  found  that  experience  to  be 
especially  problematic  in  your  organization”  and  then  pause. 
The  respondent  is  likely  to  say  something  like,  “Well,  yes,  and 
it  affected  not  only  our  organization,  but  it  had  an  impact  on 
our  relationship  with  ....” 
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Example: 

Follow-up 

questions 


The  session 
transcript 


The  questions  listed  in  the  questionnaire  are  a  guide  for  the  interview, 
but  it  will  often  be  necessary  to  follow  up  these  questions  with  probing 
or  clarifying  questions. 

Skilled  interviewers  use  several  types  of  questions  to  draw  out  the 
information  they  need.  Some  examples  of  each  are  provided  below. 


Question  type  Example 

Probing  •  “Can  you  give  me  a  specific  example  of  what 

you  mean?” 

•  “How  often  do  you  report  information  to  the 
Chief  Engineer?” 

Clarifying  •  “Do  I  understand  correctly  that  what  you’re 

saying  is  ...?” 

•  “When  you  say  that  there  are  problems  with 
communication,  what  do  you  mean?” 


The  session  transcript  is  essential  to  the  analysis  of  the  interview 
results. 

Although  you  may  have  the  capability  to  record  a  respondent  in  audio 
or  video,  most  interview  methodologists  do  not  think  that  mechanized 
recording  is  a  good  idea.  Respondents  are  often  uncomfortable  when 
they  know  their  remarks  will  be  recorded  word-for-word.  They  may 
strain  to  say  things  only  in  a  socially  acceptable  way.  Although  you 
would  get  a  more  detailed  and  accurate  record,  it  is  likely  to  be  unduly 
distorted  by  the  process  of  obtaining  it.  In  general,  personal  interviews 
are  still  best  when  recorded  by  the  interviewer  using  pen  and  paper. 
[Trochim  2001,  p.  130] 
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Guidelines: 

Recording 

responses 


Concluding  the 
interview 


•  Record  responses  immediately. 

The  scribe  should  record  responses  as  they  are  being  stated.  This 
conveys  the  idea  that  they  are  interested  enough  in  what  the 
respondent  is  saying  to  write  it  down.  The  scribe  does  not  have  to 
write  down  every  single  word,  but  certain  key  phrases  or  quotes 
should  be  recorded  verbatim.  A  system  should  be  implemented  for 
distinguishing  what  the  respondent  says  verbatim  from  what  the 
interviewers  are  characterizing. 

•  Include  all  probes. 

The  transcript  should  indicate  every  single  probe  that  is  used. 
Develop  shorthand  for  different  standard  probes.  Use  a  clear 
method  for  writing  them  (for  example,  place  probes  in  the  left 
margin).  Use  abbreviations  where  possible;  abbreviations  will  help 
interviewers  capture  more  of  the  discussion.  Develop  a 
standardized  system  (e.g.,  R=respondent;  DK=don’t  know).  If  the 
interviewer  creates  an  abbreviation  on  the  fly,  they  should  record 
its  origin.  For  instance,  if  you  as  an  interviewer  decides  to 
abbreviate  project  management  indicators  with  PMI,  make  a 
notation  in  the  right  margin  saying  PMI  =  project  management 
indicator. 


The  actions  listed  below  should  be  taken  to  bring  the  interview  to 
closure. 

•  The  interviewer  should  review  any  actions  and  issues  that 
were  identified  during  the  meeting. 

•  Immediately  after  leaving,  the  interviewer  should  write  down 
notes  about  how  the  interview  went.  Sometimes  interviewers 
will  have  observations  about  the  interview  that  they  did  not 
want  to  write  down  while  they  were  with  the  respondent. 
Immediately  after  the  interview,  go  over  the  interview  notes 
and  make  any  other  comments  and  observations,  but  be  sure 
to  distinguish  these  from  notes  made  during  the  interview  (by 
using  a  different  color  pen,  for  instance). 
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Do’s  and  Don’ts 
for  the  interviewer 


Immediately 
after  the  interview 


A  number  of  guidelines  for  conducting  the  interviews  have  been 
presented  in  this  section.  The  following  is  a  summary  list  of  important 
“do’s”  and  “don’ts”. 

Do  the  following: 

•  Focus  on  the  primary  objective  of  the  meeting  during 
information  gathering — which  is  to  collect  information. 

•  Ask  every  question.  This  will  require  discipline  on  the  part  of 
the  interviewer  who  will  need  to  manage  time  carefully.  Keep 
track  of  the  time  so  that  all  questions  are  covered  during  the 
interview  meeting.  (You  may  have  to  intervene  in  some  cases 
to  keep  the  interview  from  stalling  on  any  single  topic.) 

•  Record  responses  on  the  interview  form.  Have  a  prepared 
strategy  for  doing  this.  Do  not  rely  on  an  audio  recording 
device.  See  the  section  titled,  “Recording  the  responses”  on 
page  82  of  this  document. 

Do  not  do  the  following: 

•  Allow  personal  stories  to  take  up  valuable  time.  The  purpose  is 
to  obtain  information  from  the  interviewee. 

•  Diverge  too  far  from  the  questionnaire.  Opportunities  abound 
to  discuss  interesting  but  unrelated  matters,  but  there  is  a 
pressing  need  for  focus  given  the  time  limits. 

•  Allow  your  personal  interests  to  become  part  of  the  discussion. 
It  is  important  not  to  cue  the  respondent  with  topics  that  they 
would  not  identify  on  their  own. 


Immediately  after  the  interview  has  been  adjourned,  the  interviewer 
should  review  the  interview  transcript  and  annotate  it  as  needed  to 
clarify  any  vague  statements  that  may  be  difficult  to  interpret  at  a  later 
time  (e.g.,  abbreviations,  incomplete  thoughts).  These  annotations 
should  be  labeled  as  such  to  distinguish  them  from  the  information  that 
was  recorded  during  the  interview. 
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